Persist StateProof builders on disk #4553

Aharonee · 2022-09-15T10:10:36Z

To support indefinite StateProofs recovery period (currently only 1000 intervals are supported), we must store the date required to build a StateProof transaction on disk.
This feature is a partial implementation, migrating the existing database and adding logic for persisting and restoring StateProof builders, as well as updating the unit tests accordingly.

Persist StateProof Builders

Add flag (used by Relay nodes) to enable this feature
Migrate the StateProof database
Insert every new builder into the database
Delete entry when the relevant StateProof transaction was confirmed
Load from DB (if flag enabled) on startup

Keep less builders in memory and send less signatures

This will be implemented in a separate PR

Replace the builders map with 2 (more?) builders: NextStateProofRoundBuilder and LatestRoundBuilder.

Testing

Enable persistBuilder flag for every test to make sure nothing breaks
Write new unit tests to verify the database state
Test: the number of builders in DB is consistent with the number of pending StateProofs
Test: if stateproof is not stalled, there should only be one builder stored in DB
Fix tests in worker_test.go to timeout when waiting on txmsg channel instead of blocking indefinitely
Update unit tests to make sure number of builders in memory is consistent with that in DB (will change in the next PR when limiting memory footprint)
Write e2e tests?

…tBuilder type

… VotersForRound

…uilders from database

codecov · 2022-09-15T11:00:46Z

Codecov Report

Merging #4553 (dda383d) into feature/stateproofs-recoverability (7de989a) will increase coverage by 0.09%.
The diff coverage is 78.88%.

@@                          Coverage Diff                           @@
##           feature/stateproofs-recoverability    #4553      +/-   ##
======================================================================
+ Coverage                               55.23%   55.33%   +0.09%     
======================================================================
  Files                                     398      398              
  Lines                                   50276    50389     +113     
======================================================================
+ Hits                                    27772    27881     +109     
+ Misses                                  20188    20185       -3     
- Partials                                 2316     2323       +7

Impacted Files	Coverage Δ
node/node.go	`4.43% <ø> (ø)`
stateproof/db.go	`79.77% <74.07%> (-4.44%)`	⬇️
stateproof/builder.go	`80.14% <78.26%> (+3.22%)`	⬆️
stateproof/signer.go	`52.38% <80.00%> (+1.74%)`	⬆️
crypto/stateproof/builder.go	`88.88% <89.28%> (-0.64%)`	⬇️
stateproof/worker.go	`89.28% <100.00%> (-0.72%)`	⬇️
network/wsNetwork.go	`64.63% <0.00%> (-0.20%)`	⬇️
ledger/acctupdates.go	`69.89% <0.00%> (+0.59%)`	⬆️
catchup/service.go	`68.88% <0.00%> (+0.74%)`	⬆️
... and 3 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

…ers in memory

stateproof/worker.go

id-ms

Overall looks good.
Have some suggestions to have a clearer code.

crypto/stateproof/builder.go

stateproof/builder.go

stateproof/db.go

stateproof/builder.go

stateproof/signer.go

stateproof/worker.go

…it-tests

stateproof/builder.go

stateproof/db.go

stateproof/builder.go

stateproof/worker.go

stateproof/db_test.go

stateproof/builder.go

stateproof/db.go

Aharonee added 6 commits August 31, 2022 20:43

[WIP] Migrate stateproof database to store builders and add Persisten…

ac2cbdc

…tBuilder type

[WIP] Refactor builder and remove PersistentBuilder and dependency on…

f55e94e

… VotersForRound

[WIP] Add persistBuilders flag to stateproof.Worker, and delete old b…

8ad83d1

…uilders from database

[WIP] .

de11f79

Finalize the builder struct, fix deleteOldBuilders and make msgpack

a49b125

Finalize implementation and fix tests

1e3fed1

Aharonee self-assigned this Sep 15, 2022

Aharonee added Enhancement New Feature labels Sep 15, 2022

Aharonee force-pushed the persist_stateproof_builders branch from 49a3516 to 02887f9 Compare September 15, 2022 10:29

Aharonee requested a review from id-ms September 19, 2022 06:18

Add and update unit tests to verify database is consistent with build…

eddbfa1

…ers in memory

Aharonee force-pushed the persist_stateproof_builders branch from 02887f9 to eddbfa1 Compare September 19, 2022 06:53

Aharonee marked this pull request as ready for review September 19, 2022 06:53

algonathan reviewed Oct 6, 2022

View reviewed changes

stateproof/worker.go Outdated Show resolved Hide resolved

id-ms reviewed Oct 6, 2022

View reviewed changes

algonathan added 13 commits October 18, 2022 09:29

rename stateproof top voters alloc bound name

c576f6b

fx: removed comment

d590d6b

refactor: fetchBuilder to treat errors from decoding.

f286467

using sql.ErrNoRows

12cdf5f

rm: unused fields in stateproof/Worker

554ea13

fx: avoiding sql access for sigs in the future

d057a0d

refactor: moving persisted fields into BuilderPersistingFields struct

8f89870

fx: stating that the worker is not a persistentBuilder in numerous un…

432a0e8

…it-tests

inspect sig is in db prior to signing in func signStateProof

ee29689

improved exists claus

e679a81

reviewdog-fix

0bc918d

refactor: getBuilder func to not fill a ptr

6ea7c27

fx: fetchBuilderForRound behaviour

5553b83

algonathan added 8 commits October 19, 2022 11:29

adding unit-tests

994cde3

adding unit-test that catchs an issue

6dedf3f

fx: when fetching builder from DB - fill with sigs

9ec48e8

fx: fetchBuilderFromDB refactor

74443d2

fx:race

9a898e3

reusing code

19f3a31

fx: race

acf3d32

fx: watch-dog issue

e680324

id-ms suggested changes Oct 19, 2022

View reviewed changes

stateproof/builder.go Outdated Show resolved Hide resolved

stateproof/db.go Outdated Show resolved Hide resolved

algonathan added 4 commits October 19, 2022 16:31

fx: comment

aadb33c

rename insertBuilder func

e86c8e0

improvement: reloading sigs-per-round

172621a

removing unneeded field from func fillBuilder

ee6404e

id-ms reviewed Oct 20, 2022

View reviewed changes

algoidan added 2 commits October 20, 2022 15:28

fix CR

61b33b9

fixing edge cases with when no signatures are in the database

f2df8ef

id-ms force-pushed the persist_stateproof_builders branch from bc366d2 to f2df8ef Compare October 20, 2022 13:30

algonathan reviewed Oct 20, 2022

View reviewed changes

stateproof/builder.go Show resolved Hide resolved

algonathan reviewed Oct 20, 2022

View reviewed changes

stateproof/builder.go Show resolved Hide resolved

algonathan reviewed Oct 20, 2022

View reviewed changes

stateproof/db.go Outdated Show resolved Hide resolved

algoidan added 3 commits October 20, 2022 20:31

support old databases having only signatures

f0f1426

fix bug + work on tests

4f853d5

adjusting tests

9d78989

id-ms force-pushed the persist_stateproof_builders branch from 4179b32 to 9d78989 Compare October 21, 2022 13:47

algoidan added 4 commits October 21, 2022 17:32

close the query

d0269e6

verify that builder can be loaded with no signatures

e5ba262

remove the const flag for now

9504705

fmt

dda383d

id-ms merged commit 598705d into algorand:feature/stateproofs-recoverability Oct 23, 2022

onetechnical mentioned this pull request Mar 14, 2023

DevOps: merge master into feature/alphanet #5203

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Persist StateProof builders on disk #4553

Persist StateProof builders on disk #4553

Aharonee commented Sep 15, 2022

codecov bot commented Sep 15, 2022 •

edited

Loading

id-ms left a comment

Persist StateProof builders on disk #4553

Persist StateProof builders on disk #4553

Conversation

Aharonee commented Sep 15, 2022